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« The MAILING DA TE of this communication appears on the cover sheet with the correspondence address ~ 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )S Responsive to communication(s) filed on 30 June 2003 . 
2a)E3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Qt/ay/e, 1935 CD. 1 1, 453 O.G. 213. 
Disposition of Claims 

4) [X] Claim(s) 1-24 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 1-24 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
1 1 )□ The proposed drawing correction filed on is: a)D approved b)Q disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)0 All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2.Q Certified copies of the priority documents have been received in Application No. . 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 . 
Attachment(s) 

1 ) ED Notice of References Cited (PTO-892) 4) O Interview Summary (PTO-41 3) Paper No(s). . 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) O Notice of Informal Patent Application (PTO-152) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 



1 . This office action is responsive to the following communications: Application, 
filed 03/27/2001; Amendment A, filed 06/30/2003. 

This action is made final. 

2. Claims 1-24 are pending in this application. Claims 1 and 13 are independent 
claims. Claim 13 has been amended. 

3. The present title of this application is "Augmented Reality Guided Instrument 
Positioning with Modulated Guiding Graphics" (as originally filed). 



4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 1-24 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Fukunaga, U.S. Patent Number 6,346,940 B1, "Virtualized Endoscope System", class 
345/427, 02/2002, filed 02/1998. 



As per independent claim 1, a method for augmented reality guided instrument 
positioning, comprising the steps of: determining a graphics guide for positioning an 
instrument; Fukunaga discloses guiding marker to indicate direction of the instrument, 
virtual endoscope, col. 8, lines 1-10; and rendering the graphics guide such that an 



Claim Rejections - 35 USC § 103 
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appearance of at least one portion of the graphics guide is modulated with respect to at 
least one of space and time. Fukunaga discloses preparing a guiding marker and the 
image control unit processing the guiding marker for display at a desired position on the 
image, col. 19, lines 1-60. However, it is noted that Fukunaga fails to disclose explicitly 
modulating the graphics guide appearance. It would have been obvious to one of 
ordinary skill in the art at the time of the invention that as the virtual endoscope is 
moved about the image the graphics guide would be changed to represent the portion 
of the image the endoscope is occupying. 

With respect to dependent claim 2, rendering step comprises the step of varying 
a transparency of the at least one portion of the graphics guide with respect to other 
portions of the graphics guide to provide a substantially unobstructed view through the 
at least one portion of the graphics guide . . . Fukunaga discloses semitransparent 
image so that portions of the image can be viewed unobstructed, col. 11, lines 44-67. 
However, it is noted that Fukunaga fails to disclose a transparent guide marker. It 
would have been obvious to one of ordinary skill in the art at the time of the invention to 
include not only semi-transparent images, but also markers to avoid obstructing portions 
of the image, which is what Fukunaga wishes to accomplish. 

With respect to dependent claim 3, varying a transparency of the at least one 
portion of the graphics guide during pre-defined time intervals to provide a substantially 
unobstructed view through the at least one portion of the graphics guide . . . Fukunaga 
discloses an adjustable degree of transparency, col. 7, lines 59-64. However, it is noted 
that Fukunaga fails to disclose a transparent guide marker. It would have been obvious 
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to one of ordinary skill in the art at the time of the invention to include not only semi- 
transparent images, but also markers to avoid obstructing portions of the image, which 
is what Fukunaga wishes to accomplish. 

With respect to dependent claim 4, varying a transparency of each of a plurality 
of portions of the graphics guide during at least one predefined time interval to provide a 
substantially unobstructed view through each of the plurality of portion to at least a 
portion of the instrument . . . Fukunaga discloses an adjustable degree of transparency, 
col. 7, lines 59-64. However, it is noted that Fukunaga fails to disclose a transparent 
guide marker. It would have been obvious to one of ordinary skill in the art at the time 
of the invention to include not only semi-transparent images, but also markers to avoid 
obstructing portions of the image. 

With respect to dependent claim 5, plurality of portions are consecutive. 
Fukunaga discloses guiding markers as a path to take, col. 8, lines 5-10. 

With respect to dependent claim 6, varying a transparency of the at least one 
portion of the graphics guide such that the at least one portion repeatedly switches 
between transparent and less transparent. Fukunaga discloses thinning pixels based 
on a preset degree of semi transparency adjusted by the operator, col. 1 1 , lines 44-57. 
However, it is noted that Fukunaga fails to disclose a transparent guide marker. It 
would have been obvious to one of ordinary skill in the art at the time of the invention to 
include not only semi-transparent images, but also guide markers to avoid obstructing 
portions of the image. 
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With respect to dependent claim 7, constructing the graphics guide as a line, and 
rendering step comprises the step of modulating a transparency of the line with respect 
to time so that the line repeatedly fades in and out of view to provide a substantially 
unobstructed view . . . Fukunaga discloses the guiding marker used to indicate the 
direction could be a path, arrows, etc, col. 8, lines 1-10. However, it is noted that 
Fukunaga fails to disclose a transparent line. It would have been obvious to one of 
ordinary skill in the art at the time of the invention to include not only semi-transparent 
images, but also markers to avoid obstructing portions of the image. 

With respect to dependent claim 8, constructing the graphics guide as a line, and 
rendering step comprises the step of modulating a transparency of portions of the line 
sot that at least a portion of the instrument is substantially unobstructed . . . Fukunaga 
discloses the guiding marker used to indicate the direction could be a path, arrows, etc, 
col. 8, lines 1-10. However, it is noted that Fukunaga fails to disclose a transparent line. 
It would have been obvious to one of ordinary skill in the art at the time of the invention 
to include not only semi-transparent images, but also markers to avoid obstructing 
portions of the image. 

With respect to dependent claim 9, rendering step comprises the step of 
modulating a transparency of portions of the line with respect to time and space so that 
at least a portion of the instrument is substantially unobstructed . . . during pre-defined 
time intervals. Fukunaga discloses semitransparent image sot that portions of the 
image can be viewed unobstructed, col. 1 1 , lines 44-67. However, it is noted that 
Fukunaga fails to disclose a transparent line. It would have been obvious to one of 
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ordinary skill in the art at the time of the invention to include not only semi-transparent 
images, but also markers to avoid obstructing portions of the image. 

With respect to dependent claim 10, constructing the graphics guide as a cylinder 
. . . rendering step comprises the step of modulating a transparency of the cylinder with 
respect to time so that the cylinder repeatedly fades in and out of view . . . Fukunaga 
discloses typical examples of guiding markers. However, it is noted that Fukunaga fails 
to disclose a transparent cylinder. It would have been obvious to one of ordinary skill in 
the art at the time of the invention to include not only semi-transparent images, but also 
graphic markers to avoid obstructing portions of the image. 

With respect to dependent claim 1 1 , modulating a transparency of portions of the 
cylinder so that at least a portion of the instrument is substantially unobstructed . . . 
Fukunaga discloses semitransparent image sot that portions of the image can be 
viewed unobstructed, col. 1 1 , lines 44. However, it is noted that Fukunaga fails to 
disclose a transparent cylinder. It would have been obvious to one of ordinary skill in 
the art at the time of the invention to include not only semi-transparent images, but also 
graphic guide markers to avoid obstructing portions of the image. 

With respect to dependent claim 12, rendering step comprises the step of 
modulating a transparency of portions of the cylinder with respect to time and space so 
that at least a portion of the instrument is substantially unobstructed . . . during pre- 
defined time intervals. Fukunaga discloses a operation path history that registers the 
path position and view direction each time the instrument changes to reproduce the 
image on the display unit, col. 8, lines 32-47. However, it is noted that Fukunaga fails to 
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disclose a transparent line. It would have been obvious to one of ordinary skill in the art 
at the time of the invention to include not only semi-transparent images, but graphic also 
guide markers to avoid obstructing portions of the image. 

As per independent claim 13 and dependent claims 14-24, they are rejected 
based upon similar rational as above independent claim 1 and dependent claims 2-12 
respectively. 



Response to Arguments 

6. Applicant's arguments filed 06/30/2003 have been fully considered but they are 
not persuasive. 

Applicant argues that Fukunaga fails to disclose a graphics guide for positioning 
an instrument. Fukunaga discloses an endoscope tip state computation unit for display 
of an indicator image and the endoscope tip is the orientation, i.e. positioned, of the 
endoscope, i.e. instrument, in relation to the body and view direction, viewpoint position, 
and reference direction, col. 7, lines 12-26, and further discloses the endoscope tip is 
prepared for display either inside or outside depending on the viewpoint position, col. 7, 
lines 27-39, therefore the endoscope tip state computation unit performs the function of 
positioning an instrument. Fukunaga discloses guiding markers to indicate the three- 
dimensional coordinate positions within the image, col. 8, lines 1-10. Applicant argues 
that Fukunaga fails to disclose the graphics guide is modulated with respect to one of 
space and time. Fukunaga discloses an operation path history in which the viewpoint 
positions are indicated and the shortest path connecting the viewpoint positions is 
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determined and the path history stores the images on the main display and sub-display, 
col. 8, lines 32-57. 

Applicant further discloses that Fukunaga fails to disclose the varying portion of 
the endoscope image and varying a transparency of the graphics guide. Fukunaga 
discloses semitransparent image so that portions of the image can be viewed 
unobstructed, col. 1 1 , lines 44-67. Fukunaga discloses masking for superimposing 
images and adjusting the pattern of the masking image to change the degree of one 
image through another, col. 12, lines 10-55 and discloses the image are varied. It is 
inherent that a variation of intensity of an image may take place over a time or a 
distance. Applicant argues that Fukunaga fails to disclose a transparency of the portion 
of the graphics during pre-defined time intervals. Fukunaga discloses recorded 
simulation images, col. 5, lines 46-52, therefore portions of the image, which are viewed 
unobstructed, are transparent during pre-defined time intervals during the simulation. 

Applicant argues that Fukunaga fails to disclose modulating a transparency and 
a rendering device modulates a transparency. Fukunaga discloses varying, i.e. 
modulating, intensities of color, and a transparency parameter, col. 12, lines 10-25, 
therefore modulation is performed on the transparent color by the variation of the 
intensity of the color data. 

Conclusion 

7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Motilewa A. Good-Johnson whose telephone number is 
(703) 305-3939. The examiner can normally be reached on Monday - Friday 8:30 AM - 
5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers 
for the organization where this application or proceeding is assigned are (703) 872-9314 
for regular communications and (703) 872-9314 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 306- 
0377. 
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